#attention superposition

Fudan University researchers have developed Lorsa, a sparse attention mechanism that disentangles atomic attention units hidden in transformer superposition, enhancing interpretability of large language models.